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FOREWORD 


This  report  was  written  at  the  request  of  the  Chief  of  Naval  Operations  (OP- 11 5)  as 
an  article  summarizing  the  state-of-the-art  in  procedures  for  training  development  and 
the  role  of  NAVPERSRANDCEN  research  and  development  in  assisting  with  problems 
encountered.  It  is  intended  to  provide  manpower,  personnel,  and  training  managers  with  a 
general  overview  of  the  background  of  modem  approaches  to  training  development,  point 
out  certain  problems  existing  in  the  procedures  themselves  and  in  managing  their 
implementation,  describe  some  modest  attempts  to  improve  implementation  in  the  future, 
and  make  recommendations  for  improvement. 

This  paper  was  written  with  partial  support  from  Advanced  Development  Subproject 
Z1175-PN.05  (Improved  Effectiveness  in  Course  Design,  Delivery,  and  Evaluation)  under 
the  sponsorship  of  OP-01. 


JAMES  F.  KELLY,  JR. 
Commanding  Officer 


JAMES  J.  REGAN 
Technical  Director 


SUMMARY 


Problem  . 

The  Navy  needs  better  training  not  only  because  fleet  readiness  depends  on  it,  but 
also  because  increasing  technological  complexity  demands  it.  Fleet  readiness  depends 
upon  personnel  readiness,  which,  in  turn,  is  influenced  by  selection,  training,  job 
experience,  motivation,  leadership,  and  other  factors.  To  achieve  individual 
readness--and,  in  the  long  run,  improve  unit  and  fleet  readiness — design  technologies 
must  be  developed  and  applied  in  all  of  these  areas.  In  training,  a  design  tech¬ 
nology-instructional  Systems  Design  (ISD)--has  been  developed  and  adopted.  Although 
ISD  is  an  exciting  prospect  for  improving  the  quality  of  Navy  training  programs  and  the 
management  of  Navy  training  development,  there  are  problems  in  its  management  and 
implementation  end  in  the  adequacy  of  ISD  methods  and  procedures.  Because  of  this,  ISD 
has  not  yet  achieved  its  full  potential  for  improving  Navy  training. 

Objective 

The  purpose  of  this  report  is  to  review  the  background  of  ISD,  identify  problems  in 
the  ISD  process  and  in  its  management  and  implementation,  describe  some  research  and 
development  efforts  aimed  at  solving  these  problems,  and  recommend  additional  research 

needed. 

Approach 


Developments  leading  to  adopting  the  ISD  model  and  systematic  studies  evaluating 
the  successes  and  problems  of  ISD  implementation  were  extensively  reviewed.  From 
these  reviews,  several  research  programs  were  begun  that  aid  in  solving  certain  of  the 
problems  enountered. 

F indings  and  Conclusions 


The  ISD  process  is  a  synthesis  of  the  results  of  nearly  40  years  of  research  on 
instructional  development  and  over  100  previous  publications  concerning  what  to  do  in 
instructional  development.  The  review  found  widespread  agreement  that  the  ISD  process 
is  a  good  description  of  what  has  to  be  done  in  the  process  of  analyzing,  designing, 
developing,  and  managing  instruction. 

The  ISD  methods  were  originally  developed  to  remind  instructional  development 
experts  about  steps  needed  to  be  accomplished  to  produce  quality  instruction.  Recently, 
though,  the  intent  of  ISD  methods  has  shifted.  ISD  manuals  are  now  intended  to  help 
content  specialists  (who  are  relatively  inexperienced  in  instructional  design  and 
development)  build  instruction.  However,  ISD  methods  do  not  achieve  this  intent, 
because,  while  they  contain  "what  to  do"  information,  they  do  not  provide  information  of 
"how  to  do  it." 

This  lack  of  detailed  procedural  guidance  leads  to  two  difficulties  in  implementing 
ISD:  instructional  engineering  problems  and  management  problems.  Studies  have 
revealed  that  ISO  methods  are  not  implemented  well  by  the  untrained,  inexperienced 
personnel  available.  This  leads,  in  turn,  to  problems  in  managing  their  ISD  activity. 
These  problems  result  in  the  development  of  instruction  imperfectly  matched  to  Navy  jobs 
and  increased  costs  for  evaluation  and  revision  of  initially  poor  instruction. 
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A  number  of  policy  or  management  changes  to  alleviate  these  problems  were 
considered,  including  (1)  substitution  of  some  other  training  development  methods  in  place 
of  ISD,  (2)  substantial  training  for  ISD  practitioners  to  compensate  for  ISD  inadequacies, 
and  (3)  contracting  for  the  development  of  Navy  training  programs.  All  these  alternatives 
were  rejected  as  being  impractical,  too  expensive,  or  as  simply  cosmetic.  Instead,  it  was 
decided  that  the  best  policy  to  follow  is  to  improve  the  ISD  methods  and  their 
implementation  through  a  systematic  research  and  development  program.  Several  recent 
R&D  efforts  were  summarized,  including  the  instructional  quality  inventory  (1QI),  which 
provides  quality  assurance  methods  for  the  ISD,  the  development  of  procedures  for 
building  more  instructionally  relevant  tests  in  technical  training,  and  the  initial 
development  of  computer-based  systems  to  assist  in  managing  and  conducting  ISD. 
Computer-based  aids  to  ISD,  in  particular,  provide  the  best  hope  of  improving  the 
efficiency  and  effectiveness  of  the  ISD  process  and  are  essential  as  more  training  is 
delivered  via  computers. 


Recommendations 


i.  This  review  identified  some  deficiencies  in  the  ISD  process  and  its  implementa¬ 
tion.  However,  these  deficiencies  were  obtained  from  only  a  few  studies  of  ISD  and  from 
relatively  informal  observation.  It  is  essential  that  the  Naval  Education  and  Training 
Command  (NAVEDTRACOM)  develop  systematic  methods  for  monitoring  the  implementa¬ 
tion  of  ISD,  so  that  problems  can  be  identified.  These  problems  should  then  be  reviewed 
periodically  to  determine  their  amenability  to  management,  personnel  training,  or  R&D 
solutions. 


2.  NAVEDTRACOM  should  develop  a  system  for  monitoring  the  performance  of  ISD 
practitioners  and  managers  and  a  systematic  training  and  professional  development 

program  to  improve  their  performance. 

3.  NAVEDTRACOM  should  actively  support  the  development  of  automated  aids  to 
ISD,  and  should  begin  planning  and  programming  resource0  for  their  eventual  implementa¬ 
tion  and  operation. 
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INTRODUCTION 


Problem 

The  Navy  needs  better  training  not  only  because  fleet  readiness  depends  on  it,  but 
also  because  increasing  technological  complexity  demands  it.1  Fleet  readiness  depends 
upon  personnel  readiness,  which,  in  turn,  is  influenced  by  selection,  training,  job 
experience,  motivation,  leadership,  and  other  factors.  To  achieve  individual  readi¬ 
ness — and,  in  the  long  run,  improve  unit  and  fleet  readiness— design  technologies  must  be 
developed  and  applied  in  all  of  these  areas.  In  training,  a  design  technology— Instructional 
Systems  Design  (ISD) — has  been  developed  and  adopted.  Although  ISD  is  an  exciting 
prospect  for  improving  the  quality  of  Navy  training  programs  and  the  management  of 
Navy  training  development,  there  are  problems  in  its  management  and  implementation 
and  in  the  adequacy  of  ISD  methods  and  procedures.  Because  of  this,  ISD  has  not  yet 
achieved  its  full  potential  for  improving  Navy  training. 

Objective 


The  primary  objective  of  this  report  is  to  review  selected  problems  existing  in  the 
implementation  of  ISD  methodology  and  to  describe  briefly  some  research  and  develop¬ 
ment  projects  that  are  underway  or  needed  to  aid  future  implementation  of  the  process. 


APPROACH 

Developments  leading  to  adopting  the  ISD  model  and  systematic  studies  evaluating 
the  successes  and  problems  of  ISD  implementation  were  extensively  reviewed.  From 
these  reviews,  several  research  programs  were  begun  that  aid  in  solving  certain  of  the 
problems  enountered. 


REVIEW  OF  ISD 

Navy  curriculum  design  efforts  are  continually  being  handicapped  by  shortages  of 
experts  to  conduct  training  program  design  and  development,  poor  analysis  of  how  to 
match  training  to  jobs,  inadequate  prescriptions  for  deciding  how  to  train,  inadequate 
performance  measurements,  differences  in  student  skills  and  motivation,  problems  in 
managing  Navy  schools,  and  problems  in  planning  the  use  of  computers  and  simulators  in 
training.  It  is  important  to  recognize  that  these  problems  all  stem  from  fundamental 
inadequacies  in  our  understanding  of  how  people  learn  and,  therefore,  how  to  teach  these 
people.  This  leads  to  inadequacies  in  training  design  methods  and  procedures  and  to 
confusions  about  management  of  training  program  design  and  development.  Research  and 
development  has  led  and  will  lead  to  improvements  in  our  fundamental  understanding  of 
learning  and  instruction,  and  in  procedural  refinements  that  will  lead  to  improvements  in 
ISD  methodology  and  the  formulation  of  more  effective  management  policies.  Careful 
implementation  of  existing  knowledge  and  systematic  development  of  information  bases 
for  new  procedures  are  needed. 


lChief  of  Naval  Operations  Memorandum,  Ser  00/100622  of  11  December  1979;  subj: 
CNO  objectives  for  1980. 


The  Evolution  of  ISP 

Traditional  techniques  for  developing  instructional  programs  depend  very  heavily  on 
the  expertise  of  the  people  doing  the  development.  Until  recently,  instructional  design 
was  an  intuitive,  artistic  process.  One  problem  with  this  approach  is  that  the  intuitions  of 
people  about  what  and  how  to  teach  can  vary  widely.  The  resulting  instruction  may  then 
teach  irrelevant  things  or  perhaps  leave  out  things  that  are  very  important  to  people's  job 
performance.  A  second  problem  is  that  people's  artistry  also  varies.  While  one  artist  may 
design  instruction  that  communicates  very  efficiently,  another  person  may  produce 
"instruction'*  that  is  barely  comprehensible.  Thus,  instability  or  variability  is  built  into 
the  traditional  instructional  development  process. 

This  traditional  approach  has  further  problems.  Although  it  is  concerned  with  job 
relevancy  in  a  general  sense,  it  cannot  ensure  that  training  matches  job  requirements:  It 
might  or  might  not,  depending  on  someone's  intuition.  The  traditional  approach  also 
cannot  ensure  that  quality  training  is  developed.  Training  materials  may  be  developed 
badly,  or  materials  may  be  chosen  arbitrarily  by  an  instructor  or  developer.  Also,  in  the 
traditional  philosophy  of  instruction,  the  focus  is  usually  cm  the  content  that  is  presented 
to  the  trainee,  who  is  expected  to  learn  from  the  exposure.  Instruction  is  usually  "topic 
oriented"  in  that  it  tells  about  something  (e.g.,  how  a  radar  operates)  rather  than  being 
"performance-oriented,"  which  tells  a  student  how  to  operate  the  radar.  The  learning  is 
usually  not  evaluated  systematically,  and  training  adequacy  is  judged  in  terms  of  what 
students  say  about  it.  Little  attempt  is  made  to  identify  what  difficulty  a  student  is 
having  in  learning  and  to  correct  it.  It  is,  in  essence,  a  "sink  or  swim"  approach. 

During  the  mid  1950s,  training  developers  recognized  the  need  to  guarantee  the  job 
relevancy  and  to  monitor  systematically  inadequacies  in  learning.  They  began  developing 
techniques  to  stabilize  and  structure  the  process  of  training  development,  to  ensure  the 
relevancy  of  training  for  people's  jobs,  and  to  make  training  efficient.  This  approach, 
adapted  from  those  used  in  operations  research  and  systems  engineering  (Churchman, 
1968)  for  the  development  of  weapon  systems,  led  to  the  development  of  the  ISD  process. 

ISD  procedures  evolved  from  a  conviction  that  the  systems  analysis  approach  could 
simplify  the  complex  task  of  developing  programs  of  instruction.  Systems  analysis 
methods  had  developed  during  World  War  II  to  help  resolve  problems  in  managing  the 
design,  production,  and  evaluation  of  new  weapon  systems.  The  method  was  applied 
successfully  to  numerous  problems  whose  complexity  strained  any  one  person's  ability  to 
comprehend  and  accomplish  a  task.  For  example,  systems  analysis  has  been  given  credit 
for  the  success  of  the  project  that  put  man  on  the  moon  (Carter,  1973). 

Four  main  procedures  characterize  the  systems  approach: 

1.  A  team  of  experts  is  chosen  from  relevant  disciplines  to  bring  as  much 
information  and  expertise  as  possible  to  bear  in  accomplishing  a  task. 

2.  Models  or  simplified  descriptions  of  subparts  of  the  task  are  generated  to  reduce 
its  overall  complexity. 

3.  Unique  but  systematic  solutions  to  the  task  are  devised. 

4.  Operational  tests  are  conducted  to  provide  information  for  later  revision  and 
modification  (cf.,  Montemerlo  4c  Tennyson,  1976)  Andrews  4c  Goodson,  1980). 

As  applied  to  training  program  development,  a  group  of  experts  In  management,  logistics, 
education/training,  systems  planning,  and  other  fields  generated  model  procedures  to 


simplify  day-to-day  tasks.  For  example,  training  experts  might  devise  checklists  or  other 
outlines  to  remind  them  of  steps  in  development  that  had  to  be  completed  and  to  record 
such  steps  when  they  had  been  completed.  Although  such  procedures  help  experts 
determine  what  to  do  next,  they  do  not  supplant  the  intelligence  or  knowledge  needed  to 
carry  on  the  activity  (Montemerlo  <Jc  Tennyson,  1976;  Montemerlo,  1979a;  Andrews  Ic 
Good  son,  1980). 

During  the  1960s,  there  was  a  shift  away  from  the  systems  analysis  approach,  which 
relied  upon  teams  of  experts,  toward  development  of  formal  procedures,  models,  and 
design  decision  aids  that  would  enable  relatively  inexperienced  persons  to  design  instruc¬ 
tion.  These  procedures  and  aids  were  elaborate  forms  of  the  simple  models  and  checklists 
used  by  the  experts.  The  prospect  of  being  able  to  use  less  experienced  people  to  develop 
training  appealed  to  managers  of  military  instruction  programs  because  experts  were 
scarce — and  still  are--and  job  rotation  restricted  the  buildup  of  expertise.  Over  100 
manuals  were  published  telling  how  to  design  and  develop  programs  of  instruction  (e.g., 
Montemerlo  &  Tennyson,  1976;  Andrews  &  Goodson,  1980).  At  various  times,  these 
procedures  were  called  '^course  design  procedures,"  "curriculum  engineering,"  "systems 
approach  to  training,"  and,  more  recently,  "instructional  systems  development." 

Although  all  these  procedures  differ  in  some  details,  they  share  a  common  approach: 
They  analyze  jobs  to  determine  training  objectives,  develop  tests  to  assess  whether 
trainees  are  progressing  toward  objectives,  gear  instruction  toward  specific  learning  goals 
that  are  tied  to  the  objective,  and  attempt  to  detemine  how  to  decide  upon  the 
instructional  presentations  in  sufficient  detail  to  minimize  the  level  of  experience  needed 
in  instructional  development  and  technology. 

ISD  evolved  not  as  a  specific  method  of  teaching  but,  rather,  as  a  way  to  determine 
what  trainees  really  need  to  know  and  to  ensure  that  they  learn  it.  With  reference  to 
training,  ISO's  goals  are  to  make  training  (1)  job-relevant  (meaning  it  would  ultimately 
prepare  the  trainee  for  his  or  her  functionCs)  in  Navy  readiness),  and  (2)  cost-effective  and 
efficient  (meaning  it  would  use  the  most  efficient  training  methods  to  do  the  training). 
With  regard  to  the  management  of  the  development  of  instructional  programs,  ISD's  goals 
are  to  (1)  make  the  process  more  efficient  and  less  haphazard  and  (2)  provide  a  basis  for 
controlling  and  evaluating  changes. 

It  is  important  to  understand  these  training  and  management  goals  because  people 
have  often  confused  ISD  with  particular  new  teaching  techniques.  The  ISD  process  is  a 
means  for  defining  training  goals,  deciding  upon  the  best  means  of  achieving  them  within 
resource  constraints,  and  providing  evaluation  of  the  program.  In  theory,  any  medium  of 
presentation  (traditional,  self-paced,  or  computer-managed)  could  be  chosen,  depending 
upon  its  feasibility.  In  practice,  however,  the  recommended  method  is  self-pacing 
because  it  is  more  efficient.  Self -pacing  takes  advantage  of  the  fact  that  people  learn  at 
different  rates  (Branson,  Rayner,  Cox,  Furman,  King,  &  Harnum,  1975).  Research  has 
shown  that,  when  self-paced  courses  are  compared  with  traditional  "lock-step"  courses, 
they  save  considerable  time,  with  students  achieving  the  same  or  slightly  better  school 
performance.  For  example,  in  comparisons  done  in  48  military  training  courses, 
achievement  in  self-paced  courses  was  equal  to  that  of  conventional  courses  in  32  cases, 
superior  in  15  cases,  and  slightly  poorer  in  only  1  case  (Orlansky  ic  String,  1979).  There 
are  a  number  of  questions --which  research  can  help  answer- -about  how  to  implement 
other  types  of  individualized  instruction  and  whether  they  are  effective  for  all  trainees. 
Nevertheless,  "its  use  can  be  expected  to  continue  and  expand  in  the  Navy  as  increased 
emphasis  is  placed  on  training  efficiency,"  according  to  a  1979  report  by  the  Navy 
Training  Analysis  and  Evaluation  Group  (Zajkowski,  Heidt,  Corey,  Merv,  &  Micheli,  1979), 
unless  it  can  be  shown  that  traditional  methods  can  be  made  more  efficient  through  the 
use  of  innovative  techniques,  such  as  computer-aids  for  instructors. 
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1SD  Potentials  and  Problems 


History  shows  that  educational  innovations  in  this  century  go  through  a  peculiar 
three-stage  life  cycle  (Montemerlo  <5c  Tennyson,  1976;  Campbell,  1971).  In  the  first  stage, 
advocates  of  an  innovation  proclaim  its  usefulness  and  its  success.  In  stage  two,  many 
people  are  attracted  to  the  innovation  and  begin  using  it.  The  final  stage,  however,  is  one 
of  growing  skepticism  and  criticism  of  the  innovation's  adequacy.  Because  this  criticism 
comes  late  in  the  process,  it  does  not  help  improve  the  technique  but,  rather,  hastens  its 
abandonment  in  favor  of  yet  another  innovation.  The  process  then  begins  anew.  This  may 
explain  why  so  many  training  development  systems  differ  only  in  slight  degree  from  one 
another  (cf.,  Montemerlo  <5c  Tennyson,  1976;  Andrews  &  Goodson,  1980). 

Whatever  the  causes  of  the  cycle — and  some  theorists  believe  they  are  political  and 
social  (Milsum,  1968;  Montemerlo,  1979b)--the  effect  is  that,  when  an  educational 
innovation  is  introduced,  its  proponents  suppress  constructive  criticism  as  they  nurture 
and  protect  their  "brain  child."  The  danger  is  that  the  ISD  model,  which  has  enjoyed  its 
days  of  advocacy  and  is  now  somewhere  between  phase  two  of  widespread  use  and  phase 
three  of  growing  skepticism,  may  share  the  fate  of  so  many  other  innovations.  Yet  any 
system  that  replaces  ISD  will  present  the  same  underlying  problems.  It  will  be  new, 
unrefined,  untested,  and  probably  difficult  to  manage. 

As  ISD  moves  into  the  criticism  phase  of  its  life  cycle,  it  still  retains  much  of  its 
early  promise.  ISD  has  made  progress  in  developing  techniques  to  make  training  more  job¬ 
relevant.  As  will  be  discussed  below,  however,  problems  exist  in  successfully  implement¬ 
ing  these  techniques.  The  fact  that  some  version  of  the  systems  approach  model  has  been 
adopted  by  the  military  services,  by  industry,  and  for  training  development  in  the  civilian 
sector  suggests  that  the  approach  is  valid  (Gropper,  1980).  This  has  happened  because 
systems  approaches  all  include  the  same  general  steps  that  serve  as  a  basis  for  managing 
the  process,  and  managers  believe  that  systematic  attempts  to  relate  training  to  job 
requirements  optimize  the  use  of  resources  better  than  relatively  unsystematic  alterna¬ 
tives.  The  original  hope  that  the  procedures  of  experts  could  somehow  be  transmitted  by 
means  of  manuals  to  nonexperts,  who  then  carry  out  training  development,  does  not  seem 
to  have  been  realized  (Montemerlo  &  Tennyson,  1976;  Montemerlo,  1979a).  This  failure 
stems  from  a  less-than-adequate  state  of  knowledge  about  human  learning  and  instruction, 
as  well  as  our  inability  to  provide  recipes  for  training  development  that  untrained  people 
can  follow.  Therefore,  the  difficulties  encountered  in  using  the  ISD  process  are  of  two 
sorts:  "instructional  engineering"  and  "management"  problems.  These  problems  are 
discussed  in  the  following  paragraphs. 

Instructional  Engineering  Problems 

Current  ISD  procedure  manuals  are  supposedly  more  complete  than  earlier  ones  and 
more  relevant  to  the  development  of  training  for  a  wide  variety  of  military  tasks.  Yet,  a 
recent  study  of  the  implementation  of  ISD  by  33  groups  that  developed  57  different 
courses  in  all  the  military  services  noted  that  the  procedures  are  still  not  fully  adequate. 
In  this  study,  Vineberg  and  Joyner  (1980)  reported  that  the  job-relevancy  issue  was  often 
ignored  and  that  previously  existing  instruction  was  used  as  a  starting  point  for  course 
development.  Instructional  methods  were  selected  not  because  they  were  effective  and 
rfficient  but  because  they  existed.  Similarly,  tests  to  measure  job-related  learning  were 
limited  to  what  could  rather  than  what  should  be  tested.  Evaluation  of  training,  according 
i  to  Vineberg  and  Joyner,  received  little  emphasis.  Feedback  systems  from  operational 

(units  concerning  job  competency  of  graduates  were  not  well  developed.  As  a  result, 
.  -lining  programs  needed  extensive  tryouts  and  revision  to  make  them  effective.  These 
findings  suggest  that,  at  present,  implementation  of  ISD  has  not  succeeded  in  attaining 
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the  goals  of  making  training  job-relevant,  efficient,  or  cost-effective.  Evidence  for  this 
point  has  been  given  in  reports  of  studies  of  the  implementation  of  systems  approaches  in 
training  development.  Army  users,  for  example,  have  found  guidance  incomplete,  steps 
difficult  to  relate  to  one  another,  and  job  analyses  incomplete  (Vineberg  8c  Joyner,  1980; 
Ricketson,  Schultz,  <5c  Wright,  1970;  Hodak,  Middleton,  8c  Rankin,  1979;  Miller,  Swink,  8c 
McKenzie,  1978).  These  findings  suggest  that  those  procedures  were  not  implemented 
well  by  the  untrained,  inexperienced  personnel  available  (Montemerlo  8c  Tennyson,  1976). 

Our  analysis  of  some  Navy  courses  reveals  similar  serious  deficiencies  in  training 
objectives,  tests,  and  course  materials  (e.g.,  Wulfeck,  Stern,  Fredericks,  8c  Ellis,  1979, 
Stern  8c  Fredericks,  in  press).  Some  objectives  are  not  related  to  performance  or 
knowledge  required  by  the  job.  Testing  does  not  always  measure  performance  or 
knowledge  required  by  the  objectives.  Since  instruction  often  is  not  geared  to  the 
objectives  or  to  the  tests,  it  is  confusing  and  otherwise  inadequate.  These  problems  have 
led,  in  some  cases,  to  course  graduates  who  are  poorly  prepared  for  their  jobs,  and  this  has 
resulted  in  criticism  of  ISD  by  the  operational  community. 

How  can  it  be  that  these  problems  result  even  though  the  ISD  procedures  are 
supposedly  more  complete  than  earlier  instructional  design  systems?  It  appears  that  these 
instructional  engineering  problems  are  the  result  of  the  fact  that  the  ISD  procedures 
specify  "what  to  do"  in  detail,  but  not  "how  to  do  it,"  let  alone  "how  to  do  it  well." 

It  might  be  useful  to  underscore  this  point.  For  someone  who  is  very  knowledgeable 
and  skilled  in  a  particular  discipline,  a  simple  reminder  that  something  should  be  done  may 
be  sufficient.  This  is  why  simple  checklists  worked  well  in  the  early  attempts  to  use  a 
systems  approach  to  training  development.  In  the  development  of  procedural  guidelines, 
such  as  those  in  ISD,  the  adequacy  of  the  guidance  for  inexperienced  people  is  the  issue. 
Much  more  specific  "how  to  do  it"  information  must  be  provided. 

An  interesting  analogue  of  this  problem  was  discussed  in  a  recent  paper.  In  a 
cookbook  meant  for  experienced  chefs,  a  recipe  for  mayonnaise  was  found  to  be  six 
sentences  long  (plus  a  list  of  ingredients).  In  another  book,  intended  for  less  experienced 
cooks,  the  same  recipe  was  over  13  times  longer— some  900  words  plus  ingredients 
(Norman,  Gentner,  8c  Stevens,  1976).  Even  with  this  recipe,  a  complete  novice  in  the 
kitchen  would  have  difficulty  in  completing  it  successfully. 

Are  instructional  recipes  any  different?  Their  adequacy  depends  on  how  well  the 
persons  doing  the  work  understand  what  has  to  be  done  and  all  the  steps  in  the  process. 
To  show  the  difficulty  with  the  level  of  the  "recipes"  presented  in  ISD,  consider  that,  out 
of  a  36-page  section  on  developing  instruction,  only  two  paragraphs— four  sentences- -tell 
how  to  develop  printed  training  materials,  and  only  one  of  those  sentences  offers  real 
guidance:  "Write  all  the  student  needs  to  know  about  each  learning  objective,  but  do  not 
write  one  word  more"  (Branson  et  al.,  1975,  Vol.  Ill,  p.  246).  Does  this  provide  sufficient 
guidance  for  a  person  inexperienced  in  techniques  of  instruction  and  communication? 

In  addition  to  the  lack  of  adequate  "how  to  do  it"  information,  the  ISD  guidance  also 
contains  internal  inconsistencies.  For  example,  the  ISD  process  requires  that  developers 
(1)  specify  objectives,  (2)  develop  tests,  (3)  sequence  the  objectives  into  modules,  lessons, 
etc.,  and  (4)  develop  instruction.  The  problem  is  that,  although  test  items  can  be 
generated  from  objectives,  they  cannot  be  arranged  into  modules  or  lesson  tests  until  the 
objectives  have  been  sequenced.  Therefore,  things  are  out  of  order.  The  guidelines  say  to 
construct  tests  before  we  know  what  they  should  contain.  Similar  inconsistencies  can  be 
found  throughout  the  ISD  manuals. 
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Management  Problems 

These  fundamental  problems  in  the  adequacy  of  ISD  guidance  lead  to  problems  in 
managing  the  ISD  process.  The  management  problems  are  further  complicated  by  the  fact 
that  managers  may  not  be  aware  of  them.  Training  development  managers  often  treat  the 
ISD  model  and  procedures  as  if  they  were  complete  and  adequate  and  assume  that,  if  a 
developer  merely  follows  "the  book,"  instruction  will  be  adequate.  There  is  no  recognition 
of  the  variability  caused  by  the  inadequate  procedures.  Since  the  ISD  model  contains  no 
methods  for  assuring  the  quality  of  materials  as  they  are  developed,  managers  may  pay 
little  or  no  attention  to  that  quality.  They  do  not  know  how.  Rather,  they  consider  the 
development  of  training  to  resemble  assembly-line  production;  thus,  the  documentation  of 
each  step  in  the  process  receives  more  emphasis  than  the  way  the  inadequate  guidelines 
affect  instructional  quality.  This  trivialization  of  the  process  can  and  does  result  in 
inadequate  programs  that  need  numerous  revisions  to  make  them  job-relevant.  Managers 
tend  to  be  satisfied  when  all  tasks  and  forms  are  completed,  even  though  students  may 
falter  (e.g.,  Montemerlo,  1979a;  Vineberg  Sc  Joyner,  1980;  Hodak,  Middleton,  Sc  Rankin, 
1979;  Miller,  Swink,  Sc  McKenzie,  1978;  Middleton,  Zajkowski,  Sc  Aagard,  1979). 

Other  more  apparent  management  problems  include  selecting  and  training  design  and 
development  personnel,  financing,  and  other  administrative  and  logistics  issues.  In  the 
area  of  selecting  and  training  personnel,  the  failure  to  spell  out  procedures  in  sufficient 
detail  makes  it  necessary  to  find  or  extensively  train  people  with  expertise  in  training 
technology.  Costs  are  further  driven  up  as  each  new  instructional  program  requires  (1) 
new  people  to  learn  how  to  develop  it  and  (2)  extensive  revisions  to  validate  it.  The 
decision  by  the  Naval  Education  and  Training  Command  (NAVEDTRACOM)  to  establish  a 
central  organization  for  developing  training  programs  reflects  a  tacit  recognition  that 
some  expertise  is  needed  to  compensate  for  incomplete  procedures.  The  Navy's 
Instructional  Program  Development  Centers  (IPDCs)  include  civilian  technologists  who  are 
expected  to  have  or  develop  and  maintain  expertise.  Most  course  development  and 
revision  will  take  place  outside  these  centers  for  some  time,  however.  Building  up 
expertise  for  regular  duty  Navy  personnel  is  difficult  because  of  job  rotation  policies  and 
the  lack  of  appropriate  occupational  specialties  in  instructional  technology.  Navy  people 
do  provide  subject-matter  expertise  and,  despite  their  lack  of  appreciable  training  in 
instructional  development  per  se,  are  given  responsibility  for  developing  advanced 
courses.  In  this  case,  the  quality  of  the  instruction  depends  on  the  intuitions  of  the 
personnel.  It  is  not  necessarily  bad,  but  its  quality  is  uncertain. 

It  has  recently  become  clear  that  the  costs  of  adopting  the  ISD  model  are  high. 
However,  instructional  development  costs  may  always  have  been  high.  In  the  past,  these 
costs  were  "hidden"  in  the  normal  assignment  of  instructors  to  Navy  schools.  To  illustrate 
this  point,  in  the  Navy  and  in  the  other  services,  25  percent  of  an  instructor's  workweek  is 
accounted  for  by  categorizing  it  as  course  preparation  or  administrative  duties.  There¬ 
fore,  traditionally,  course  development  or  preparation  costs  are  high  even  if  only  half  that 
time  is  considered  to  be  course  development.  In  addition,  actual  course  development  was 
and  still  is  done  within  Navy  schools.  In  a  survey  completed  several  years  ago,  it  was 
found  that,  for  a  2  1/2  year  period,  four  to  seven  Instructors  had  been  assigned  to  develop 
objectives  for  a  new  course.  They  were  considered  part  of  the  school's  complement  of 
instructors.  The  adoption  of  the  IPD  approach  has  made  costs  more  apparent  because  of 
the  way  in  which  costs  are  reported,  and  not  necessarily  because  costs  actually  have  been 
increased.  2»* 


’Chief  of  Naval  Education  and  Training  Instruction  1550.5  of  11  March  1974;  subj: 

Doctrine  for  the  centralized  management  and  control  of  ISD. 

’Chief  of  Naval  Education  and  Training  Support  Code  N4,  Direct  Cost  Report,  of  30 
June  1979. 

£  _ _ _ _  .  A 
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APPROACHES  FOR  IMPROVING  THE  ISD  MODEL  AND  IMPLEMENTATION 


Policy  Considerations 

As  we  have  seen,  fundamental  instructional  engineering  problems  in  ISD  create 
problems  in  the  implementation  and  management  of  instructional  development  programs 
in  the  Navy.  What  policy  alternatives  are  available  to  solve  these  problems? 

One  alternative  could  be  to  discard  the  current  ISD  methodology  and  "start  from 
scratch."  As  stated  earlier,  however,  all  systematic  approaches  to  instructional  develop¬ 
ment  include  the  same  general  steps.  As  a  "what-to-do"  procedure,  ISD  is  generally 
adequate,  and  anything  that  replaced  it  would  be  fundamentally  the  same. 


A  second  alternative  would  be  to  leave  ISD  as  it  is  but  increase  the  training  of  ISD 
practitioners  so  that  they  could  compensate  for  the  inadequacies  in  the  process.  It  would 
be  necessary  to  determine  the  amount  of  training  developers  need  and  to  provide  it  on  a 
continuing  basis  for  both  Navy  civilian  and  military  personnel.  NAVEDTRACOM's  policy 
of  developing  courses  in  IPDCs  is  intended  to  build  up  the  level  of  expertise.  These 
centers,  however,  concentrate  on  preliminary  technical  training,  and  apprentice-level 
courses,  and  their  expertise  will  not  generally  be  available  to  the  personnel  developing  and 
modifying  the  thousands  of  other  Navy  courses.  Wider  dispersion  of  training  and  other 
means  of  assisting  development  of  these  courses  would  be  needed.  There  are  some  serious 
limitations  in  giving  training  in  instructional  technology  as  a  solution  to  this  problem. 
Since  the  knowledge  base  for  the  ISD  procedures  is  incomplete,  training  for  ISD 
practitioners  would  be  difficult  to  specify  and  develop.  It  would  probably  have  to  be 
broad  in  scope  and  lengthy — probably  equivalent  to  a  graduate  program  in  educational 
technology— and  would  have  to  be  offered  on  a  continuing  basis  because  of  personnel 
turnover.  The  impracticality  and  expense  of  such  a  program  is  obvious.  Naturally,  under 
any  conditions,  some  in-service  training  will  have  to  be  offered  on  a  continuing  basis,  and 
systematic  evaluation  of  people's  performance  is  necessary  to  assess  their  skill  and  to 
maintain  their  competency.  Such  training  and  evaluation,  however,  will  not  be  sufficient 
to  compensate  for  ISD  inadequacies. 


Another  alternative  is  for  the  Navy  to  abandon  the  curriculum  development  business 
completely  and  contract  for  the  development  of  training  programs.  This  approach  has 
serious  limitations  since  potential  contractors  (1)  have  no  better  procedures  for  course 
design  than  the  Navy  does,  (2)  are  not  sufficiently  familiar  with  Navy  operational  and 
training  situations  to  develop  courses  that  fully  meet  Navy  requirements,  (3)  may  not 
exist  in  sufficient  numbers  to  handle  the  course  development  requirements,  and  (4)  may 
cause  implementation  problems  resulting  from  the  "hand-off"  process  from  contractor  to 
Navy  personnel. 


The  policy  alternative  that  seems  most  appropriate  is  to  retain  current  ISD 
methodology  in  its  general  form,  but  to  actively  support  a  program  of  research  and 
development  aimed  at  refining  it,  augmenting  it  where  necessary,  and  improving  its 
implementation  and  management.  The  need  for  systematic  refinement  of  ISD  methods 
and  their  implementation  has  already  been  recognized.  The  Chief  of  Naval  Education  and 
Training  and  NAVPERSRANDCEN  have  agreed  to  identify  and  seek  R&D  solutions  to 
problems  that  impede  successful  implementation  of  the  ISD  process.1* 


"Memorandum  of  Agreement  between  NAVPERSRANDCEN  and  CNET  on  research 
requirements,  implementing  R&D,  and  establishing  the  Experimental  Training  Programs 
Policy  Board,  ser  284, 13  May  1975. 
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Improving  ISP  Through  Research  and  Development 


The  main  deficiency  of  ISO  procedures  discussed  so  far  is  their  failure  to  allow 
relatively  inexperienced  personnel  to  develop  reliably  good  instruction  (Hodak,  Middleton, 
&  Rankin,  1979;  Miller,  Swink,  ic  McKenzie,  1978).  This  section  describes  some  research 
and  development  efforts  designed  to  refine  and  augment  the  ISO  methodology  through  (1) 
conducting  R&D  aimed  at  acquiring  the  knowledge  base  necessary  to  develop  more  usable 
and  manageable  ISD  techniques,  (2)  improving  the  current  process  by  developing 
procedures  to  fill  ISD  gaps  (e.g.,  quality  control  techniques  and  test  development 
methodologies),  and  (3)  improving  the  implementation  and  management  of  ISD  by 
capitalizing  on  computer  technology. 

The  Instructional  Quality  Inventory  (IQI) 

After  examining  the  ISD  process,  it  was  recognized  that,  at  several  points  in  the 
developmental  sequence,  intermediate  products  (e.g.,  objectives,  test  items,  segments  of 
instruction,  etc.)  were  available,  but  there  was  no  check  on  their  quality.  This  is 
particularly  unfortunate,  since  later  steps  in  the  ISD  sequence  depend  on  the  quality  of 
these  products.  Therefore,  the  development  of  techniques  for  quality  assurance  during 
ISD  was  undertaken.  First,  the  research  literature  was  examined  to  determine  what  the 
tested  or  valid  principles  for  prescribing  instruction  were.  Then,  techniques  were  devised 
to  allow  an  evaluator  to  examine  instruction  and  determine  whether  or  not  it  conformed 
to  those  prescriptions.  In  brief,  the  evaluator  makes  sure  that  (1)  the  stated  objectives 
are  job-relevant,  (2)  test  items  are  adequate  for  the  purpose  of  assessing  learning  and  are 
congruent  with  the  objectives,  and  (3)  course  content  is  matched  appropriately  to  the 
objectives  and  thereby  to  the  job  requirements.  This  technique  is  called  the  Instructional 
Quality  Inventory  (IQI). 

A  preliminary  version  of  the  IQI  was  developed  by  Professor  M.  D.  Merrill  and 
associates  under  contract  to  NAVPERSRANDCEN  (Merrill  &  Boutwell,  1973;  Merrill  & 
Wood,  1974;  Merrill,  Richards,  Schmidt,  &  Wood,  1977).  The  IQI  then  underwent  extensive 
testing  and  revision  in  Navy  courses  to  ensure  that  the  procedures  were  clear  and  could  be 
used  by  those  who  would  be  doing  instructional  development.  User  manuals,  training 
materials,  and  job-aids  were  published  (Wulfeck,  Ellis,  Richards,  Merrill,  &  Wood,  1978; 
Ellis,  Wulfeck,  &  Fredericks,  1979;  Ellis  &  Wulfeck,  1978).  The  Chief  of  Naval  Education 
and  Training  has  since  recommended  their  use  throughout  the  N AVEDTR ACOM.  *  ’* 

In  helping  make  instructional  materials  and  tests  consistent  with  learning  objectives, 
the  IQI  fills  important  gaps  in  ISD  procedures  that  were  revealed  in  surveys  of 
instructional  development  personnel  and  of  ISD  implementation  (Vineberg  &  Joyner,  1980; 
Hodak  et  al,  1979;  Middleton  et  al,  1979).  Until  now,  no  systematic  means  existed  for 
determining  whether  existing  instruction  is  adequate  for  current  purposes,  thereby 
avoiding  unnecessary  development  costs.  ISD  did  require  that  existing  instruction  be 
reviewed  but  provided  no  guidance  for  this.  As  a  result,  this  step  received  little  attention 
in  the  development  of  new  courses  (Vineberg  &  Joyner,  1980).  The  IQI  procedure  can  also 
be  used  to  examine  the  quality  of  contractor -developed  Instruction. 


'Chief  of  Naval  Education  and  Training,  CNETNOTE  1530  of  6  April  1979;  subj: 
Instructional  Quality  Inventory  (IQI). 

'Chief  of  Naval  Education  and  Training,  CNETINST  1550.15  of  29  May  1980;  subj: 
Instructional  Program  Development  Centers  QPDCs);  policy  and  guidance  for. 


Guidelines  for  Criterion-referenced  Testing 

One  of  the  goals  of  1SD  is  to  make  training  relevant  to  job-performance  require¬ 
ments.  Such  performance-based  training  requires  techniques  for  testing  different  from 
those  familiar  to  most  people.  The  tests  commonly  encountered  are  those  that  are  used 
to  compare  a  person's  test  performance  with  that  of  others.  Examples  of  such  tests  are 
those  used  to  select  people  for  admission  to  training,  advancement-in-rate,  or  college 
entrance,  or  to  rank  people  relative  to  some  group  as  in  IQ  tests,  Civil  Service 
professional  administration  career  examinations,  and  the  Armed  Forces  Qualification 
Test.  These  tests  are  called  "norm-referenced,"  meaning  that  an  individual's  test 
performance  is  compared --referenced — to  the  average  or  "norm"  of  everyone  else's  test 
scores.  Normative  tests  are,  at  best,  only  indirectly  related  to  people's  jobs.  In  contrast, 
performance-based  training  requires  tests  that  directly  measure  aspects  of  performance 
required  in  course  objectives.  Tests  of  this  type  are  called  "criterion-referenced," 
meaning  that  an  individual's  test  performance  is  compared— referenced— to  an  absolute 
objective  or  criterion  indicating  what  a  person  must  know  or  be  able  to  do.  Quite 
different  problems  are  confronted  in  developing  and  using  these  tests  in  contrast  to  norm- 
referenced  tests. 

In  addition  to  determining  that  an  individual  can  do  or  does  know  whatever  is 
specified  in  the  objectives,  it  is  necessary  to  be  able  to  determine  why  a  trainee  cannot 
perform  well  so  that  the  problem  or  errors  can  be  corrected.  Whenever  remedies  are 
prescribed  on  the  basis  of  test  results,  diagnostic  tests  are  needed.  Diagnostic  tests  must 
be  designed  to  give  further  information  about  gaps  in  student  knowledge  or  skill,  just  as  a 
physician  uses  additional  diagnostic  tests  when  symptoms  indicate  a  medical  problem.  For 
example,  consider  a  situation  in  which  a  student  does  not  understand  some  instruction  he 
has  just  read  concerning  when  to  use  a  particular  procedure.  When  tested  regarding  when 
to  use  it,  he  makes  an  error.  Since  this  is  the  only  information  the  instructor  has,  the 
cause  of  the  error  is  unknown.  Usually,  the  student  is  given  the  same  material  to  reread. 
If  the  misunderstanding  is  due  to  the  fact  that  the  explanation  is  unclear  to  the  student, 
many  rereadings  of  the  instruction  will  not  correct  the  error.  Techniques  for  identifying 
why  students  make  such  errors  are  diagnostic.  Once  the  misunderstanding  is  diagnosed,  it 
can  be  corrected  quickly. 

Although  both  criterion-referenced  and  diagnostic  tests  are  needed  in  ISD,  guidelines 
are  incomplete  for  developing  the  former  and  nonexistent  for  developing  the  latter. 
Therefore,  improved  procedures  for  test  development  in  ISD  are  being  developed.  The 
resulting  "handbook"  will  compile  procedures  from  diverse  sources  (Merrill  &  Wood,  1974; 
Swezey  &  Pearlstein,  1974;  Roid  &  Haladyna,  1979;  Frederickson,  Smith,  &  Pearlstein, 
1979)  and  arrange  them  as  "job-aids"  for  Navy  test  developers.  The  handbook  is  currently 
being  refined  during  training  workshops  for  Navy  personnel  and  a  final  version  will  be 
provided  to  NAVEDTRACOM  in  FY  1982.  The  handbook  will  provide  better  guidance  for 
test  development  throughout  the  NAVEDTRACOM.  It  will  provide  more  complete  and 
more  usable  procedures  than  now  exist. 

Computer-assisted  Training  Development 

The  earlier  description  of  the  ISD  process  gave  no  indication  of  record-keeping 
requirements,  although  formidable  record-keeping  problems  exist.  A  typical  Navy 
training  program  has  hundreds  or  sometimes  even  thousands  of  learning  objectives  that 
must  be  developed,  cross-referenced,  tested,  and  taught.  For  example,  about  7,000 
learning  objectives  are  contained  in  the  training  for  P-3  aircraft  crews  (Daubek, 
Freedman,  Walker,  &  Thode,  in  press). 
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Records  also  cover  a  wide  variety  of  other  ISO  activities  such  as  generating  test 
items,  choosing  alternative  training  media  and  strategies,  evaluating  graduates,  and 
revising  courses. 

Computerized  data  management  systems  not  only  can  assist  with  these  record¬ 
keeping  problems  but  also  facilitate  the  development  process  itself  by  providing  guidance 
for  test  and  instructional  development  and  similar  tasks.  Moreover,  computer-based 
systems  can  ensure  that  guidance  is  followed  by  monitoring  and  evaluating  developers' 
performance  and  by  assisting  them  as  they  proceed.  Computer  systems  can  also  provide 
training  for  instructional  developers,  who  can  fit  it  into  their  work  schedules. 

NAVPERSRANDCEN  is  now  developing  computer-based  aids  for  instructional  design 
and  development  and  will  evaluate  their  usefulness.  The  first  phase  will  help  designers  at 
critical  points  in  the  ISD  process  by  guiding  and  monitoring  each  step  involved  in  using  the 
IQI  and  developing  tests.  Then  programs  will  be  expanded  gradually  to  provide  specific 
guidance  on  accomplishing  each  task,  allow  access  to  relevant  data  bases  (e.g.,  test-item 
files,  classifications  of  objectives,  etc.),  and  help  select  alternative  forms  of  instruction. 
These  aids  will  include  computerized  ISD  data  management. 

Some  computer-based  aids  will  be  adapted  from  those  being  developed  by  other 
agencies.  For  example,  the  Army  Research  Institute  is  developing  aids  to  help  authors 
write,  edit,  and  produce  training  materials  (Schultz,  Hibbits,  Wagner,  &  Seidel,  1979).  The 
intent  is  to  make  the  production  of  materials  simpler,  faster,  and  less  costly.  Another 
system  developed  by  the  Navy  Training  Analysis  and  Evaluation  Group  (TAEG)  (Braby, 
1979;  Braby,  Parrish,  Guitard,  &  Aagard,  1978)  can  be  used  by  authors  who  are  subject- 
matter  specialists  to  write,  edit,  and  produce  programmed  training  materials.  The 
authors  use  a  terminal  that  is  connected  to  an  editing  and  production  system.  An 
executive  program  requests  information  about  materials  and  test  items  that  the  author 
provides  by  typing  on  a  keyboard.  The  system  then  arranges  the  material  in  a  particular 
format  and  produces  a  programmed  text.  Time  required  for  authoring  and  producing 
materials  is  reduced  and,  more  important,  requirements  for  instructional  design  expertise 
are  reduced.  Such  devices  enable  personnel  with  only  modest  experience  in  instruction  to 
develop  quality  instructional  materials. 

TAEG  has  also  developed  a  computer  readability  and  editing  system  (CRES),  which  is 
designed  to  improve  the  ease  of  comprehending  Navy  technical  manuals  and  training 
materials.  The  system  has  features  to  detect  uncommon  and  misspelled  words  and  long 
sentences,  suggest  simple  replacements  for  difficult  words,  and  calculate  the  readability 
grade  level.  Each  feature  is  consistent  with  Navy  specifications  and  has  been  tested  to 
verify  that  it  provides  useful  feedback  to  editors  and  authors  (Kincaid,  Aagard,  &  O'Hara, 

1980) .  CRES  has  recently  been  extended  to  incorporate  some  of  the  IQI  criteria,  so  that 
it  assists  authors  in  building  better  tests  in  technical  training  (Kincaid,  Braby,  &  Wulfeck, 

1981) .  All  of  these  systems  are  prototypes  of  more  advanced  ones  that  will  design  as  well 
as  produce  instructional  materials. 

The  systems  described  so  far  are  intended  mainly  to  support  development  of 
conventional  training  materials.  However,  as  most  people  are  aware,  there  is  a  current 
revolution  in  computer  technology  for  training  delivery.  Computers  are  getting  smaller, 
more  powerful,  and  much  less  expensive.  There  is  also  a  revolution  in  the  areas  of 
artificial  intelligence  and  cognitive  science  that  is  less  visible  but  perhaps  more  exciting. 
New  techniques  are  being  developed  for  giving  computers  real  knowledge,  for  interacting 
with  students  in  a  more  tutorial  fashion,  and  for  improving  the  Interface  between  the 
student  and  the  instructional  device.  These  developments  have  the  potential  for  making 
training  better,  cheaper,  more  manageable,  and  more  widely  distributed. 
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As  attractive  as  these  developments  are,  it  cannot  be  assumed  that  they  will 
immediately  revolutionize  training.  Currently,  the  ISD  manuals  contain  no  information 
concerning  computer-based  instructional  delivery.  Yet,  if  such  devices  are  to  be  widely 
used,  methods  for  designing  and  developing  instructional  software  must  be  institu¬ 
tionalized.  The  introduction  of  computers  into  the  ISD  process  complicates  it,  because 
additional  attention  must  be  given  during  the  analysis  and  development  phases  to 
instructional  logic  (which  the  instructor  normally  does),  planning  the  student-computer 
interaction  and  interface,  the  types  of  student  response  data  required,  the  schoolhouse 
utilization  and  maintainability  of  the  hardware  and  software,  and  a  variety  of  other 
issues.  Computer-based  author  aids,  then,  must  be  developed  to  support  ISD  for  new  types 
of  instructional  delivery. 


CONCLUSIONS 

The  ISD  process  is  a  synthesis  of  the  results  of  nearly  40  years  of  research  on 
instructional  development  and  over  100  previous  publications  concerning  what  to  do  in 
instructional  development.  The  review  found  widespread  agreement  that  the  ISD  process 
is  a  good  description  of  what  has  to  be  done  in  the  process  of  analyzing,  designing, 
developing,  and  managing  instruction. 

The  ISD  methods  were  originally  developed  to  remind  instructional  development 
experts  about  steps  needed  to  be  accomplished  to  produce  quality  instruction.  Recently, 
though,  the  intent  of  ISD  methods  has  shifted.  ISD  manuals  are  now  intended  to  help 
content  specialists  (who  are  relatively  inexperienced  in  instructional  design  and 
development)  build  instruction.  However,  ISD  methods  do  not  achieve  this  intent, 
because,  while  they  contain  "what  to  do"  information,  they  do  not  provide  information  on 
"how  to  do  it." 

This  lack  of  detailed  procedural  guidance  leads  to  two  difficulties  in  implementing 
ISD:  instructional  engineering  problems  and  management  problems.  Studies  have 

revealed  that  ISD  methods  are  not  implemented  well  by  the  untrained,  inexperienced 
personnel  available.  This  leads,  in  turn,  to  problems  in  managing  their  ISD  activity. 
These  problems  result  in  the  development  of  instruction  imperfectly  matched  to  Navy  jobs 
and  increased  costs  for  evaluation  and  revision  of  initially  poor  instruction. 

A  number  of  policy  or  management  changes  to  alleviate  these  problems  were 
considered,  including  (1)  substitution  of  some  other  training  development  methods  in  place 
of  ISD,  (2)  substantial  training  for  ISD  practitioners  to  compensate  for  ISD  inadequacies, 
and  (3)  contracting  for  the  development  of  Navy  training  programs.  All  these  alternatives 
were  rejected  as  being  impractical,  too  expensive,  or  as  simply  cosmetic.  Instead,  it  was 
decided  that  the  best  policy  to  follow  is  to  improve  the  ISD  methods  and  their 
implementation  through  a  systematic  research  and  development  program.  Several  recent 
R&D  efforts  were  summarized,  including  the  IQI,  which  provides  quality  assurance 
methods  for  the  ISD,  the  development  of  procedures  for  building  more  instructionally 
relevant  tests  in  technical  training,  and  the  initial  development  of  computer-based 
systems  to  assist  in  managing  and  conducting  ISD.  Computer-based  aids  to  ISD,  in 
particular,  provide  the  best  hope  of  improving  the  efficiency  and  effectiveness  of  the  ISD 
process,  and  are  essential  as  more  training  is  delivered  via  computers. 
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RECOMMENDATIONS 


1.  This  review  identified  some  deficiencies  in  the  ISD  process  and  its  implementa¬ 
tion.  However,  these  deficiencies  were  obtained  from  only  a  few  studies  of  ISD  and  from 
relatively  informal  observation.  It  is  essential  that  NAVEDTRACOM  develop  systematic 
methods  for  monitoring  the  implementation  of  ISD,  so  that  problems  can  be  identified. 
These  problems  should  then  be  reviewed  periodically  to  determine  their  amenability  to 
management,  personnel  training,  or  R&D  solutions. 

2.  NAVEDTRACOM  should  develop  a  system  for  monitoring  the  performance  of  ISD 
practitioners  and  managers  and  a  systematic  training  and  professional  development 
program  to  improve  their  performance. 

3.  NAVEDTRACOM  should  actively  support  the  development  of  automated  aids  to 
ISD  and  should  begin  planning  and  programming  resources  for  their  eventual  implementa¬ 
tion  and  operation. 
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